Method and Apparatus for Determining a Road Condition

ABSTRACT

This invention relates to a method and apparatus for determining a road condition using a vehicle camera ( 6 ), and it includes the following steps:
         at least one image (I) is taken using the vehicle camera (S 10 );   a first image area (R 1 ) is determined that includes an image of the road surface ( 1 ) (S 16 );   said first image area (R 1 ) is fed to a classifier, wherein the classifier assigns at least one class to said first image area that represents a specific road condition (S 18 ); and   information is output with respect to this at least one road condition (S 20 ).

This invention relates to a method and apparatus for determining a roadcondition using a vehicle camera.

Technological progress in the field of optical image acquisition allowsthe use of camera-based driver assistance systems which are locatedbehind the windshield and capture the area in front of the vehicle inthe way the driver perceives it. The functionality of these assistancesystems ranges from automatic headlights to the detection and display ofspeed limits, lane departure warnings, and imminent collision warnings.

Starting from just picking up the area in front of the vehicle to a full360° panoramic view, cameras can be found in various applications anddifferent functions for driver assistance systems in modern vehicles. Itis the primary task of digital camera image processing as an independentsource of sensor data or in conjunction with laser or lidar sensor datato detect, classify, and track objects in the image area. Classicobjects typically include various vehicles such as cars, trucks,two-wheel vehicles, or pedestrians. In addition, cameras detect trafficsigns, lane markings, guardrails, free spaces, or other generic objects.

Automatic learning and detection of object categories and theirinstances is one of the most important tasks of digital image processingand represents the current state of the art.

Modern driver assistance systems use various sensors including videocameras to capture the area in front of the car as accurately androbustly as possible. This environmental information, together withdriving dynamics information from the vehicle (e.g. from inertiasensors) provide a good impression of the current driving state of thevehicle and the entire driving situation. This information can be usedto derive the criticality of driving situations and to initiate therespective driver information/alerts or driving dynamic interventionsthrough the brake and steering system.

Since the actually available friction coefficient or equivalentinformation about the current road condition is typically not providedor cannot be measured or determined in driver assistance systems thatare ready for series production, the times for issuing an alert or forintervention are in principle determined based on a dry road with a highadhesion coefficient between the tire and the road surface.

This results in the following fundamental problem. Accident-preventingor at least impact-weakening systems warn the driver or intervene solate that accidents are prevented or accident impacts acceptablyweakened only if the road is really dry. But the effect of driverdynamic interventions via the brake and steering system is criticallydependent on the friction coefficient of the ground. Moisture, snow, andice reduce the coefficient of friction available between the tire andthe road considerably compared to a dry road. If the road provides lessadhesion due to moisture, snow, or even ice, an accident can no longerbe prevented and the reduction of the impact of the accident does nothave the desired effect.

A known approach to counteracting this fundamental problem is toevaluate camera images for the purpose of estimating road conditions andfor deriving estimated friction coefficients.

Document DE 10 2004 018 088 A1 discloses a road recognition systemhaving a temperature sensor, an ultrasound sensor, and a camera. Thetemperature, roughness, and image data (road data) obtained from thesensors is filtered and compared to reference data, and a margin ofsafety is created for the comparison. The condition of the road surfaceis determined based on the comparison of the filtered road data with thereference data. The road surface (e.g. concrete, asphalt, dirt, grass,sand, or gravel) and its condition (e.g. dry, icy, snow-covered, wet)can be classified in this way.

Document WO 2012/110030 A2 discloses a method and apparatus forestimating the friction coefficient using a 3D camera, such as a stereocamera. At least one image of the vehicle environment is taken using the3D camera. A height profile of the road surface is created in the entirearea in front of the vehicle from the image data of the 3-D camera. Theanticipated local coefficient of friction of the road surface in thearea in front of the vehicle is estimated from the height profile. Inindividual cases, classification of the road surface as snow cover or amuddy dirt road can be based on specially determined elevation profiles.

However, the known methods make high demands on the required sensors.The methods or apparatuses mentioned require a temperature and anultrasound sensor in addition to a camera, or a camera that isconfigured as a 3D sensor, to obtain sufficiently robust classificationresults.

It is therefore the object of the present invention to provide a roadcondition detection system using a camera which ensures reliable androbust proactive road condition detection or the coefficient of frictionestimation derived from it, respectively, using (only) a single monocamera.

The following considerations outline the starting point of the solutionaccording to the invention:

Algorithms of digital image processing linked with intelligentadjustment and alignment of the image area or ROI (region of interest)that is relevant for processing to the respective driving situation isto ensure that the image area analyzed contains the road surface for thepurpose of determining the road condition.

A central idea of the invention, from the point of view of digital imageprocessing, is the calculation of local and global features from theimage area (ROI), the appropriate combination of various features withinan image area or from different image areas, and the subsequent decisionby a classifier that can be trained using sample data, the findings ofwhich from various time periods result in a decision about the roadcondition. The technological advantage lies in more efficient processingof the camera image due to simple operations and in achieving a highquality through the combination of various features.

A method according to the invention for determining a road conditionusing a vehicle camera includes the following steps:

-   -   at least one image is taken using the vehicle camera;    -   a first area that includes a depiction of the road surface is        determined in the image taken;    -   said first image area is fed to a classifier, wherein the        classifier assigns at least one class to said first image area        that represents a specific road condition; and    -   information is output with respect to this at least one road        condition.

The vehicle camera captures an environment outside the vehicle; inparticular, the camera may be directed toward the front and disposedbehind the windshield near the interior rear view mirror.

The first image area that is determined for detecting the road conditionmay also be called a region of interest (ROI) and include the entireimage or a section thereof. The image area may for example be a simplerectangle, a region of an undefined shape, or even a single pixel.Determination of the image section that is relevant for further imageprocessing is particularly important to ensure that the analyzed firstimage area includes the road surface, allowing to determine the roadcondition from said first image area.

A classifier (or classification system) assigns at least one of thepredetermined classes of road conditions to the first image area. Theseclasses preferably are “wet road”, “dry road”, “snow-covered road”, and“icy road”. The classifier may in particular have been trained usingsample data. Based on learned assignments of sample image areas to knownroad conditions, the trained classifier can assign as yet unknown imagecontents or areas to at least one class.

Information is output about the at least one road condition, preferablyto other driver assistance functions, vehicle functions, or to thedriver.

The information output for the determined road condition may inparticular be an estimate of the coefficient of friction for the roadsection depicted in the image area. The friction coefficient, alsocalled coefficient of friction, adhesion coefficient, or frictionfactor, indicates the maximum force in relation to the wheel load thatcan be transmitted between a road surface and a vehicle tire (e. g. inthe tangential direction) and is therefore an essential parameter of theroad condition. In addition to the road condition, tire properties arerequired for a complete determination of the friction coefficient. It istypically just road condition information that is considered for anestimate of the friction coefficient from camera image data, since tireproperties generally cannot be detected from camera image data.

The method according to the invention for determining the road conditionensures a very robust, reliable, and proactive detection of thespatially resolved road condition. Automatic capture of road conditiondata is a key element on the road towards highly automated or autonomousdriving in the future.

In an advantageous embodiment, at least one feature is extracted fromthe first image area and fed to the classifier. The feature(s) is/areparticularly suited for detecting the different appearance of thepavement in the camera image depending on the road condition.

Several individual features may form a feature vector that combinesvarious data from the first image area to be able to make a more robustand accurate decision about the road condition in the classificationstep. Various feature types for an image area make up a set of featurevectors. The set of feature vectors obtained in this way for one imagearea is called the feature descriptor. Where multiple image areas areused, the feature descriptor may also be composed or combined ofcombined features of the various image areas. The feature descriptor canbe assembled by simple concatenation, a weighted combination, or othernon-linear representations. Not only various image areas at one point intime in an image but across several points in time in subsequent imagesof a series of images may be used. The feature descriptor is thenassigned to at least one class by a classification system (classifier).A classifier in this case is a mapping of the feature descriptor on adiscrete number that identifies the classes to be detected.

The feature that is extracted from the first image area and fed to theclassifier preferably includes the mean gray scale value or mean colorvalue (RGB) of the first image area. The “mean RGB color value” featuretype includes three individual features or feature values, namely R, G,and B (red, green, and blue value), which can be combined into a featurevector.

But any other information that can be extracted from a ROI or frompixels of the ROI and from which differences between the predeterminedclasses can be determined is suitable.

Preferably, HSI values (hue, saturation, intensity) averaged as featuretypes across the first image area or L*a*b* values (CIELAB color space)or gradient values can be extracted as features. The feature vectors forsingle or multiple feature types that are extracted from one or moreROIs of an image form the feature descriptor.

Advantageously, the at least one feature that is extracted from thefirst image area and fed to the classifier includes the result(s) of apixel-by-pixel segmentation within the first image area. Special regionscan be localized with pin-point accuracy within an image area. This isan advantage for detecting local differences, such as for detectingpuddles, drying lanes on a wet road, or icy lanes on snow-covered roads.This increases the quality of detection for these conditions. Pinpointaccurate classification may for example be achieved using semanticsegmentation method in which each pixel in the image area is assigned alabel of one of the predetermined classes. Pixel-precise classificationof images adds pinpoint accurate classification to the roughlocalization of objects in images.

According to an advantageous embodiment, a random decision forest (orjust ‘random forest’) is used as a classifier.

Decision trees are hierarchical classifiers which break down theclassification problem in a tree-like fashion. Starting at the root, apath towards a leaf node where the final classification decision is madeis followed based on previous decisions. Due to the high learningcomplexity, very simple classifiers, so-called ‘decision stumps’, whichseparate the input parameter space orthogonally to a coordinate axis,are preferred for the inner nodes.

Decision forests are collections of decision trees which containrandomized elements preferably at two points in the training of thedecision trees. First, every tree is trained with a random selection oftraining data, and second, only one selection of permissible dimensionsis used for each binary decision. Class histograms are stored in theleaf nodes which allow a maximum likelihood estimation with respect tothe feature vectors that reach the leaf node during the training. Classhistograms store the frequency with which a feature descriptor of aspecific road condition reaches the respective leaf node while travelingthrough the decision tree. As a result, each class can preferably beassigned a probability that is calculated from the class histograms.

To make a decision about a road condition for a feature descriptor, themost probable class from the class histogram is preferably used as thecurrent road condition. Other methods may also be used to transferinformation from the decision trees into a decision about the roadcondition.

According to a preferred embodiment, the assignment of the first imagearea to at least one class by the classifier for at least one imagetaken is subjected to temporal filtering before the information aboutthe at least one assigned road condition is output. The classifierassigns at least one class to an image taken or an image area thereof.An optimization step may follow this assignment or decision per imagetaken. This optimization can in particular take the temporal contextinto account in that it functions as temporal filtering. The assignmentfor the current image taken is compared to road conditions assigned toprevious images. In particular, the most frequent class from a previoustime period can be used as reference. Individual outliers(misallocations) can be eliminated in this way.

The temporal filtering advantageously includes that the assignment ofthe first image area to at least one class by the classifier for atleast one image currently taken is compared with an assignment based onat least one previously taken image. A change of the assigned roadcondition class is output only if a probability assigned to the change,which is derived from the classification of the image currently taken,exceeds a threshold value.

The temporal context is preferably taken into account by using aso-called hysteresis threshold value method. The hysteresis thresholdvalue method uses threshold values to control the change from one roadcondition into another. A change is made only when the probability ofthe new road condition is high enough and the probability of the oldroad condition is accordingly low. This stabilizes the classificationresult and prevents permanent jumping between various road conditions.

Alternatively, or cumulatively to temporal filtering, other informationfrom the vehicle, such as from the rain sensor or other data provided bythe vehicle can be used to check the assignment by the classifier beforeinformation about the at least one assigned road condition is output.

In a preferred embodiment, the position, size, and/or shape of the firstimage area is adjusted to the current driving situation of one's ownvehicle. Orientation (in the image currently taken) and tracking (inimages subsequently taken) of at least one image area that is adjustedin shape, size, and position to the driving situation is preferablyperformed taking into account the movement of one's own vehicle,potential other road users, and the road conditions.

Orientation and tracking of an image area adjusted in shape, size, andposition to the driving situation are in particular performed asfollows:

-   a) The first image area is the overall camera image if the camera is    exclusively directed towards the road.-   b) The first image area is at least one fixed area which through    adjustment and calibration of the vehicle camera is projected onto    the road in front of the vehicle, preferably in a central position    or in front of the left or right vehicle wheels.-   c) The first image area is at least one dynamic image section which    in the image is projected into the vehicle path that is calculated,    inter alia, from odometry data of the vehicle and tracked    dynamically.-   d) The first image area is at least one dynamic image section which    in the image is projected into the road/lane on/in which the vehicle    is traveling and which is within two or next to one traffic lane    boundary line and tracked dynamically.-   e) The first image area is at least one dynamic image section which    in the image is projected into the road/lane on/in which the vehicle    is traveling and which is detected by means of digital image    processing and tracked dynamically.-   f) The first image area is at least one dynamic image section which    in the image is projected into the estimated course of the road and    tracked dynamically.-   g) The first image area is at least one dynamic image section which    in the image is projected into the trajectory calculated by the    system, preferably as the center line of a predicted driving route    based on a predictive trajectory planning method, and tracked    dynamically.-   h) The first image area is at least one dynamic image section which    is projected in the direction of travel in front of the vehicle    based on GPS vehicle data, preferably in accordance with the vehicle    speed and heading angle (or yaw angle), and tracked dynamically.-   i) The first image area is at least one dynamic image section which    is projected in the direction of travel in front of the vehicle    based on vehicle odometry data and tracked dynamically.-   j) The first image area is at least one dynamic image section which    is projected onto the road in the direction of travel in front of    the vehicle based on vehicle position and map data and tracked    dynamically.-   k) The first image area is at least one fixed or dynamic image    section which corresponds to the intersection of individual areas    when at least two areas a) to j) overlap.-   l) The first image area is at least a fixed or dynamic image section    containing an area consisting of a) to k), wherein image segments    with detected objects such as vehicles, pedestrians, or    infrastructure are excluded.

Advantageously, the adjustment may be performed as a function of thespeed of one's own vehicle. The position, size, and/or shape of thesecond image area is preferably adjusted to the speed of one's ownvehicle to obtain an evenly timed prediction of the anticipated roadcondition. For example, the system can determine which road condition ispassed over in 0.5 seconds or in one second.

The required estimation of the distance can be performed with sufficientaccuracy by means of image geometry even with a mono camera if theinstallation height is known and a flat course of the road is assumed.When using a stereo camera, the distance can be determined moreprecisely by triangulation.

Advantageously, a traffic lane is detected in which the vehicle islocated and the first image area is adjusted to include an image of theroad surface of one's own traffic lane lying ahead.

In particular, a system for detecting traffic lane markings may beprovided, and the at least one “dynamic” image area includes theroad/lane on which the vehicle is traveling and which is located betweentwo or next to one traffic lane boundary line. The size of the firstimage area is preferably limited in the lateral direction by trafficlane markings or boundaries. The shape of the first image area may bethat of a trapezoid or a rectangle.

This image area can be projected into images taken subsequently, takinginto account odometry and time information, such that this image area istracked dynamically.

Odometry information includes information that characterizes themovement of the vehicle, in particular vehicle sensor data such asmeasured variables of a chassis, drive train, steering system, ornavigation device of a vehicle. Together with temporal information,movement along a distance or a trajectory of the vehicle can bedetermined.

In a preferred embodiment, a trajectory of one's own vehicle ispredicted and a vehicle path is calculated. The basis of this predictionmay be data from the camera, other environmental sensors, vehiclesensors, navigation devices, telematics equipment or the like. The firstimage area is adjusted such that it includes an image of the roadsurface that is in the calculated vehicle path.

It is particularly preferred that the first image area is adjusted suchthat the first image area only contains an image of the road surface.Relevant in this context is everything that the tires of one's ownvehicle will pass over or could pass over. Typically, the following aredeemed relevant, for example: Road surface, precipitation on it,contamination (leaves, paper, sand, oil, roadkill), traffic lanemarkings the car passes over.

The following are typically irrelevant, for example: continuous trafficlane boundary lines, grass cover to the side of the road.

The first image area can advantageously be adjusted to exclude imagesegments with previously detected objects from said first image area.Previously detected objects are, in particular, other road users, suchas vehicles (cars and trucks), two-wheeled vehicles, pedestrians, orinfrastructure elements.

It is preferred that navigation and map data and/or vehicle sensor dataand/or other environmental sensor data are taken into account whenadjusting the first image area.

According to an advantageous further development of the invention, asecond image area is determined that includes an image of a second areaof the road surface.

For example, the first image area may correspond to a predicted vehiclepath area in which the left vehicle wheels will roll on the road surfaceand the second image area may be a predicted vehicle path area in whichthe right vehicle wheels will roll.

The use of two image areas rather than one extended image area has theadvantages of requiring less computing power and time for imageprocessing than for just a single image area that includes both separateimage areas, and of providing a higher spatial resolution for theclassification of the road condition. Local changes in road conditionssuch as snow-free tracks on snow-covered roads, as are often seen inScandinavia, iced over puddles or the like can be detected and takeninto account more accurately when the image is split up into smallerimage areas.

Three or more such image areas can be determined.

The second image area advantageously includes an image of the section ofthe road surface that lies further ahead. A preferred embodiment couldtherefore include two image areas, wherein the first image area is inthe ego lane directly in front of the ego vehicle and a second imagearea is positioned in the same lane further ahead of the vehicle basedon the vehicle speed.

The size of both image areas is preferably limited in the lateraldirection by traffic lane markings or boundaries as described above.

It is preferred that the first and the second image areas do not overlapand can be spatially separate from one another. The second image area isin particular analyzed in the same way as the first image area inaccordance with the steps described above. A separate second imagesection provides the advantage of higher spatial resolution compared toan enlarged single image section.

The position, size, and/or shape of the second image area is preferablyadjusted to the speed of one's own vehicle to obtain an evenly timedprediction (or preview) of the anticipated road condition.

Advantageously, assignment of the first image area to at least one roadcondition from a currently taken image is checked for plausibility byassigning the second image area to at least one road condition from apreviously taken image. Information is therefore output about at leastone plausibility checked road condition. Since the second image areaincludes an image of a section of the road surface that lies furtherahead, its classification practically provides a preview. In a laterimage, the area of the road surface is at least partially in the firstimage area because the vehicle moves forward. The former class of thesecond image area may be taken into account as a preview forplausibility checking when the first image area that is immediatelydecisive for the further travel is classified. This increases thereliability of detection. The two image areas are preferably transformedupon one another using odometry data of the vehicle.

In a preferred embodiment, assignment of the second image area to atleast one road condition from a currently or previously taken image ismerged with the assignment of the first image area to at least one roadcondition from the currently taken image and information about themerged road condition is output.

Advantageously, a monocular camera is used as the camera. Mono camerasare established as driver assistance cameras and are more cost efficientthan stereo cameras. The method according to the invention allows robustand reliable classification of road conditions based on mono cameraimages.

Alternatively, a 3D or stereo camera is used in another advantageousembodiment. 3D or stereo cameras allow the analysis of depth informationfrom the image. In addition, the 3D position data can be used for easierharmonization of a past trajectory determination from odometry and timeinformation with the image data or for easier inclusion of a calculatedfuture trajectory or predicted vehicle route into the image data.Furthermore, depth information profiles can be used for classification.

The invention further relates to an apparatus for determining a roadcondition including a vehicle camera, an image processing unit, aclassification unit, and an output unit.

The vehicle camera is configured to take at least one image of thevehicle surroundings. The image processing unit is configured todetermine a first image area that includes an image of the road surfaceand to feed it to the classification unit. The classification unit isconfigured to assign the first image area to at least one class thatrepresents a specific road condition. The output unit is configured tooutput information about the at least one road condition which theclassification unit assigned to the first image area.

The invention will be explained in more detail below with reference tofigures and exemplary embodiments.

Wherein:

FIG. 1 is a flow chart to illustrate the flow of an embodiment of themethod for determining a road condition using a vehicle camera;

FIG. 2 is an image of a vehicle environment lying ahead that wasrecorded using a vehicle camera;

FIG. 3 is a representation of the scene depicted by the image from abird's-eye view;

FIG. 4 is an image with a first image area;

FIG. 5 is an image with a first image area and a second image areaoffset from it;

FIG. 6 is a representation for determining a predictive adaptationhorizon;

FIG. 7 is the current and future course of a trajectory while drivingalong a bend;

FIG. 8 is an image with a first image area and an image area offset fromit taking into account the course of the traffic lane;

FIG. 9 is a comparison of the current ACTUAL and the calculated PLANNEDtrajectory in an avoiding maneuver; and

FIG. 10 is an image with a first image area and two image areas offsetfrom it taking into account the predicted vehicle path.

FIG. 1 shows a flow chart to illustrate the flow of an embodiment of themethod according to the invention for determining a road condition usinga vehicle camera.

First, an image is taken using the vehicle camera in step S10. Thisimage can be used in step S12 to detect the road, e.g. based on trafficlane markings in the image, lane boundary objects, etc. Non-stationaryobjects that should not be taken into account when determining the roadcondition may be detected as early as early as in this step.

Optionally, the trajectory or path of one's own vehicle may be predictedin step S14. Data from the vehicle's own sensors (V), e.g. steeringangle, speed, etc. navigation system data or map data (N), or data fromother environmental sensors such as radar, lidar, telematics unit, etc.may be taken into account here.

In step S16, the ROI or a first image area or several image areas thatinclude(s) an image of the road surface is/are determined. This or theseimage section(s) or features extracted therefrom are fed to a classifierin step S18 which assigns each image area to at least one class thatrepresents a specific road condition.

In step S20, information about this at least one road condition isoutput, e.g. to a collision warning system or an emergency brakeassistant which can adjust its warning thresholds or intervention timesto the road condition determined.

FIG. 2 shows an example of an image (I) of the vehicle environment lyingahead as taken from the front camera (6) of a moving vehicle.Camera-based driver assistance functionality can be implemented from thesame image, e. g. a lane departure warning (LDW) function, a lanekeeping assistance/system (LKA/LKS), a traffic sign recognition (TSR)function, an intelligent headlamp control (IHC) function, a forwardcollision warning (FCW) function, a precipitation detection function, anadaptive cruise control (ACC) function, a parking assistance function,an automatic emergency brake assist (EBA) function or emergency steeringassist (ESA) function.

The camera image shows a road (1) whose surface is substantiallyhomogeneous. Traffic lane markings are visible on the surface: onecontinuous sideline (4) on each side which mark the left and right endof the road and median strip segments (3) of the broken or dashed roadmarking. The road (1) could be formed of asphalt or concrete. A puddle(2) is visible on the otherwise dry road (1).

FIG. 3 is a representation of the scene depicted by the image of thevehicle camera in FIG. 2 from a bird's-eye view. This representation canbe determined from the camera image, wherein, if a mono camera is used,imaging properties of the camera (4), the installation geometry of thecamera in the vehicle (5), the actual vehicle height (due to tireposition/chassis control), pitch angle, yaw angle, and/or roll angle arepreferably considered. It may be assumed that the road surface is even.

When a 3D or stereo camera is used, the representation can be determinedimmediately from the captured 3D image data, wherein other aspects maybe taken into consideration as well.

The representation is generally characterized in that the distancescorrespond to actual distances. The median strip segments are disposedequally spaced on the real road.

In the representation shown in FIG. 3, the road (1), the puddle (2), themedian segments (3), and the continuous lateral boundaries (4) of theroad marking that are already contained in the camera image (FIG. 2) arevisible. In addition, the representation contains a vehicle (5) with acamera (6), and said camera (6) was used to take the image from FIG. 2.FIG. 3 shows a coordinate system in which the X direction corresponds tothe longitudinal direction of the vehicle and the Y directioncorresponds to the transverse direction of the vehicle. Vehicle or realspace coordinates are labeled with uppercase letters.

The dashed arrow indicates the predicted trajectory (T) of the vehicle(5). For this driving straight ahead, the distance traveled s along thetrajectory (T) in the X direction can be determined in the case ofuniform motion based on the velocity v taking into account theinformation about the time t from s=vt. It could be determined in thisway based on odometry and time information when, for example, the leftfront wheel of the vehicle (5) will reach the puddle (2).

Determination of the Region of Interest (ROI) of a Given Camera ImageDepending on the Respective Driving Situation

Starting from a given camera image (I), we will hereinafter call thatimage area the region of interest (ROI) that contains the highestinformation density with respect to the road condition for specificdownstream functionalities (ACC, road condition estimator, etc.) FIG. 4shows such an image area (R1) within the camera image (I).

The center of a first image area (R1) assumed to be a rectangle isdescribed by the image coordinates (x₀, y₀) and the dimension of (Δx₀,Δy₀) in the entire camera image (I). Image coordinates are labeled withlowercase letters. It is possible to increase the information contentfor downstream control systems through proper adjustment of the firstimage area (R1) to the respective driving condition of the ego vehicle(5).

FIGS. 5 and 7 show a second image area (R2) in addition to the firstimage area (R1). This can be analyzed simultaneously with the firstimage area (R1) (that is, for a single image (I) to be analyzed). Thefirst image area (R1) contains information about the road condition thatis reached by the vehicle (5) in a shorter period of time, and thesecond image area (R2) contains information that will become relevant ata later time (preview for the current first image area).

Alternatively, the second image area (R2) can illustrate an adjustmentof the first image area (R1) to a faster vehicle speed (or other changeddriving situations).

As an example of such an adjustment of the first image area (R1), we usean adjustment based on the vehicle's own speed, the course of thetraffic lane while driving through a bend, and the predicted vehiclepath in an avoiding maneuver.

1. Adjustment of the First Image Area (R1) to the Speed of One's OwnVehicle (Driving Straight Ahead)

In the drive straight ahead shown in FIG. 5, the first image area (R1)could for example indicate the section of the road (1) that the vehicletravels in is at a speed of 50 km/h. If the vehicle goes twice as fast,the area of the road (1) that the second image area (R2) depicts wouldbe passed over in one second. As the vehicle speed increases, the ROI(R2) moves further into the upper portion of the image (farther awayfrom the vehicle (5)) and slightly to the left due to the cameraperspective (x₁₀<x₀, y₁₀>y₀) while its dimensions get smaller (Δx₁₀<Δx₀,Δy₁₀<Δy₀).

FIG. 6 illustrates the determination of this adjustment using arepresentation in the vehicle coordinate system (X, Y). From the pointof view of one's own vehicle (5) whose center of gravity CoG_(Veh) is ata current position X_(0Veh), a predictive adaptation horizon determinedwhich is a function of the vehicle speed V_(Veh) and optionally of otherenvironmental information Inf_(Umf):

X _(pVeh) =f(v _(Veh) ,Inf _(Umf))

When the vehicle (5) moves straight ahead, the predictive adaptationhorizon X_(pVeh) is moved in the positive X direction relative to theactual vehicle position X_(0Veh) since the vehicle moves in itslongitudinal direction (see FIG.).

The environmental information Inf_(Umf) could for example indicate thatan image area (R1, R2) should be adjusted further such that a vehiclemoving in front (not shown) is not depicted in this image area. Thiscould result in a faulty classification of the road condition. Toprevent this, an image area (R1, R2) should be reduced, cropped, ormoved in this situation such that it only depicts the road surface (1,2) to be classified.

A suitable algorithm then performs the transformation of the determinedprediction horizon (X_(pVeh)) into the image coordinate system (x, y) todetermine the new position (x₁₀, y₁₀) and dimensions (Δx₁₀, Δy₁₀) of theadjusted or changed image area. The transformation corresponds to thetransition from a representation as in FIG. 3 to a representation as inFIG. 2.

2. Adjustment of the First Image Area (R1) to a Course of Traffic LaneLying Ahead (in this Example, Passing Through a Bend)

Traffic lane markings (3, 4) can be detected from an image (I) of thevehicle camera (6) and for example used for the lane departure warningfunction (LDW). If the course of the traffic lane markings (3, 4) isknown, the course of the traffic lane the vehicle (5) moves in can bedetermined.

FIG. 7 shows the current and predicted trajectory (T) as determined by atraffic lane detection function in the vehicle coordinate system whilepassing through a bend. The mean curvature κ_(C) of the predictedtrajectory (dashed line T) can be indicated as a function of the actualvehicle yaw movement κ_(act) of the current trajectory (continuous lineT) and of additional environmental information, in particular of thecurvature of the course of traffic lane lying ahead.

$\kappa_{C} = {f\left( {{\kappa_{act} = \frac{{\overset{.}{\Psi}}_{Veh}}{\nu_{Veh}}},{Inf}_{Umf}} \right)}$

FIG. 8 shows for right-hand traffic how a prediction horizon determinedfrom FIG. 7 (not shown there) can be transformed in the image (I) byadjusting the image area from R1 to R2. In the left-hand bend shownhere, the ROI moves into the top left camera image corner (x₂₀<x₀,y₂₀>y₀). The area of the image area diminished in accordance with thecamera image. A trapezoid shape positioned on the road between themedian (3) and the right-hand traffic lane boundary line (4) wasselected for the two depicted image areas (R1, R2).

3. Adjustment of the First Image Area (R1) to a Predicted Vehicle Pathin an Avoiding Maneuver

The vehicle path is the predicted corridor of movement of the egovehicle (5) up to about 150 m distance. It is in particularcharacterized by kits width, which may match the width of the trafficlane. The vehicle path can be calculated from camera data and data ofother environmental or vehicle sensors.

If the respective camera and environmental sensors and monitoring of thedriver's activities warrant an avoiding maneuver, the area of the roadsurface (1) to be depicted by the ROI is moved based on an optimallyplanned avoiding trajectory (e.g. of the second order).

$Y = {{f(X)} = \begin{Bmatrix}{{{\frac{2 \cdot S_{Y}}{S_{X}^{2}} \cdot x^{2}},}\mspace{194mu}} & {X \leq \frac{S_{X}}{2}} \\{{{- S_{Y}} + {\frac{4S_{Y}}{S_{X}} \cdot X} - {\frac{2 \cdot S_{Y}}{S_{X}^{2}} \cdot X^{2}}},} & {X > \frac{S_{X}}{2}}\end{Bmatrix}}$

The required lateral vehicle offset is labeled S_(Y), the availableavoidance space in the X direction is S_(X). The curvature of theoptimum avoidance curve κ_(ref) is derived from the planned avoidingtrajectory according to:

${\kappa_{ref}(X)} = {\frac{1}{R} = \left| \frac{f^{n}(X)}{\left( {1 + {f^{\prime}(X)}^{2}} \right)^{\frac{3}{2}}} \right|}$

The currently driven curvature κ_(act) is a function of the vehicle yaw(see Section 2). The prediction horizon X_(pVeh) is a measure of the“forward planning” with which individual points (X_(pVeh), Y_(pVeh)) ofthe optimum avoidance curve become the target image coordinates (x₃₀,y₃₀) for the ROI.

FIG. 9 shows a previous trajectory T_(ist) in the vehicle coordinatesystem that would result in a collision with an obstacle (7) if it werecontinued. An optimum avoiding trajectory T_(soll) is shown as adashed-dotted line. It could have been used to circumvent the obstaclecomfortably. During an emergency maneuver, a trajectory as shown in thedashed-dotted line may be required due to a short-term change in yawangle from Ψ_(act) to Ψ_(ref) to enter the range of the plannedtrajectory as effectively as possible.

In such an emergency maneuver, however, determining the road conditionor camera-based estimation of the friction coefficient is extremelyimportant since the brake and steering system brakes or steers up to thelimit of the friction coefficient. A puddle (2) on an otherwise dry road(1) as shown in FIG. 2 could mean that a collision with the obstaclecannot be avoided or that one's own vehicle leaves the road. FIG. 10shows a camera image (I) depicting a stationary obstacle (7), e.g. avehicle, in the traffic lane used by the ego vehicle (6). It shows inaddition to the calculated vehicle path (or corridor of movement) T withthe continuous median trajectory and the dotted sidelines for anavoiding maneuver how a prediction horizon X_(pVeh), Y_(pVeh) determinedfrom FIG. 9 can be transformed in the image (I) by adjusting the imagearea from R1 to R1″. An intermediate step of the adjustment (R1′) isalso shown.

In the case shown (avoidance to the left), the position and adjustmentof the first image area R1′ to R1″ after appropriate transformation ofthe movement variables (X,Y) into the image coordinate system (x,y)results in a rectangular image area R1″ at position (x₃₀<x₀, y₃₀>y₀)with the dimensions (Δx₃₀, Δy₃₀). This would ensure that a change inroad condition or friction coefficient would be detected or reliablyestimated and taken into account when performing the emergency maneuver.

LIST OF REFERENCE SYMBOLS

-   1 Road or road surface-   2 Puddle-   3 median line segment-   4 continuous traffic lane boundary line-   5 (own) vehicle-   6 Vehicle camera-   7 Obstacle-   S10 Image-taking-   S12 Road detection-   S14 Trajectory planning-   S16 ROI determination-   S18 ROI classification-   S20 Output of the road condition-   V Vehicle sensor data-   N Navigation/map data-   E Environmental sensor data-   I Image-   T Trajectory (of movement)-   X Longitudinal vehicle coordinate-   Y Transverse vehicle coordinate-   x,y Image coordinates-   R1 First image area/ROI-   R2 Second or adjusted image area/ROI-   CoG_(Veh) Center of gravity of the vehicle-   X_(0Veh) Y_(0Veh) Current X position or Y position of the vehicle-   X_(pVeh) Y_(pVeh) Position of the predictive adaptation horizon-   v_(Veh) Vehicle speed-   Inf_(Umf) Environmental information-   M Center of the curve circle-   κ_(act) Actual curvature-   κ_(C) Predicted curvature-   Ψ_(act) Actual yaw angle-   {dot over (Ψ)}_(Veh) Actual yaw rate-   T_(ist) Actual trajectory-   T_(soll) Planned trajectory-   κ_(ref) Curvature of the planned trajectory-   Ψ_(ref) Yaw angle that results in the planned trajectory

1-20. (canceled)
 21. A method for determining a road condition using avehicle camera (6) of a subject vehicle, comprising: acquiring at leastone image (I) with the vehicle camera (S10); determining, in the image,a first image area (R1) that includes an image of a road surface (1)(S16); providing said first image area (R1) to a classifier, and withthe classifier assigning said first image area to at least one classthat respectively represents at least one specific road condition (S18);and outputting information regarding the at least one specific roadcondition (S20).
 22. The method according to claim 21, furthercomprising extracting at least one feature from the first image area(R1) and feeding the at least one feature to the classifier.
 23. Themethod according to claim 22, wherein the at least one feature includesa mean gray scale value or a mean color value of the first image area(R1).
 24. The method according to claim 22, wherein the at least onefeature includes results of a pixel-by-pixel segmentation within thefirst image area (R1).
 25. The method according to claim 21, wherein theclassifier comprises a random decision forest.
 26. The method accordingto claim 21, further comprising subjecting the assignment of the firstimage area (R1) to the at least one class by the classifier (S18) totemporal filtering before the outputting of the information regardingthe at least one specific road condition (S20).
 27. The method accordingto claim 26, wherein the temporal filtering comprises comparing theassignment of the first image area (R1) to the at least one class by theclassifier for a current image with a previous assignment based on atleast one previously acquired image, and outputting a change of theinformation regarding the at least one road condition only if aprobability assigned to the change, which is derived from theclassification of the current image, exceeds a threshold value.
 28. Themethod according to claim 21, further comprising adjusting a position,size, and/or shape of the first image area (R1) based on a currentdriving situation of the subject vehicle.
 29. The method according toclaim 28, wherein the current driving situation comprises a speed of thesubject vehicle (5).
 30. The method according to claim 28, furthercomprising detecting a traffic lane in which the subject vehicle islocated, and adjusting the first image area (R1) to include an image ofthe road surface (1) of the traffic lane lying ahead of the subjectvehicle.
 31. The method according to claim 28, further comprisingpredicting a trajectory (T) of the subject vehicle (5) and calculating avehicle path, and adjusting the first image area (R1) to include animage of the road surface (1) of the vehicle path.
 32. The methodaccording to claim 28, wherein the first image area (R1) is adjustedsuch that the first image area contains only an image of the roadsurface (1).
 33. The method according to claim 28, wherein the firstimage area (R1) is adjusted to exclude, from said first image area,image segments having previously detected objects.
 34. The methodaccording to claim 28, wherein navigation and map data (N) and/orvehicle sensor data (V) and/or other environmental sensor data (E) aretaken into account when adjusting the first image area (R1).
 35. Themethod according to claim 21, further comprising determining a secondimage area (R2) that includes an image of a section of the road surface(1) that lies further ahead than the first image area, wherein the firstand second image areas (R1; R2) do not overlap, and further comprisingperforming said providing and said assigning with respect to said secondimage area similarly as with respect to said first image area.
 36. Themethod according to claim 35, further comprising checking a plausibilityof the assignment of the first image area (R1) from a currently acquiredimage to said at least one class representing said specific roadcondition, by comparing the assignment of the second image area (R2)from a previously acquired image to at least one class representing aspecific road condition, and outputting the information regarding atleast one plausibility-checked road condition.
 37. The method accordingto claim 35, further comprising merging the assignment of the secondimage area (R2) from a currently or previously acquired image (I) tosaid class representing said specific road condition, with theassignment of the first image area (R1) from the currently acquiredimage to said class representing said specific road condition, andoutputting the information regarding at least one merged road condition.38. The method according to claim 21, wherein the vehicle camera (6) isa mono camera.
 39. The method according to claim 21, wherein the vehiclecamera (6) is a 3D camera or a stereo camera.
 40. An apparatus fordetermining a road condition comprising: a vehicle camera (6) that isconfigured to acquire at least one image (I) (S10), an image processingunit that is configured to determine, in the image, a first image area(R1) that includes an image of a road surface (1) (S16), aclassification unit that is arranged to receive the first image areafrom the image processing unit, and that is configured to assign thefirst image area to at least one class that respectively represents atleast one specific road condition (S18), and an output unit that isconfigured to output information regarding the at least one specificroad condition (S20).